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SANTRAUKA 

Kokybes uztikrinimo ir jvertinimo tyrimai, kaip ir kiti socialiniai 
tyrimai bei jg taikymas, kelia daugybg problemg. Siame straips- 
nyje pirmiausiai noreciau pateikti tokig problemg s^rasq, ku- 
ris vis delto liks nepilnas. Straipsnyje teigiama, kad nors ir ne- 
ra „tobulo“ sig problerruj sprendimo budo, kritinis multipliz- 
mas galetg buti toks poziurig modelis, kuris leistg sumazinti 
tas problemas arba bent padetg jas issiaiskinti ir meginti sprgs- 
ti. Kritinis multiplizmas - tai bendro pobudzio metodologinis 
modelis, kuri nuodugniai isaiskinti cia butg nejmanoma; nore- 
ciau aptarti tik kai kuriuos svarbiausius klausimus ir remda- 
masis jais pateikti keletq pasiulymg, kaip turetg buti atliekami 
socialiniai tyrimai. 

PAGRINDINiy SAVOKy APIBREZIMAI 

• Diskriminantinis pagrjstumas- kai du skirtingg savybig vertini- 
mai tais paciais ar skirtingais metodais duoda skirtingus rezulta- 
tus ir tuo parodo, jog tos dvi savybes is tikrgjg yra skirtingos. 

• Gabumas- „Bet kuri asmens savybe, kuri prognozuoja sek- 
mes tikimybg taikant tarn tikr^ poveikj. (...) Asmenybe kaip ir 
gebejimas jtakojantis reagavimq j tarn tikrq mokymq.“ (Cron- 
bach, Snow, 1981 , p.5). 

• jvedinimas- praktinio konteksto analize mokslinemis priemo- 
nemis, kuriomis siekiama surasti sprendimus tame kontekste; 
(1) institucijg /Ve/ft/wnas- tyrimas, ar institucija pasiekia iskeltus 
tikslus ir kaip jos darbas galetg buti patobulintas; (2) programu 
jvertinimas- tyrimas, ar tarn tikra programa tinkama pasiekti is- 
keltus tikslus ir kokiomis sqlygomis ji tinkama naudoti. 

• jvertinimo tyrimas (1) - jvertinimo arba jvertinimo studijg ty- 
rimas; (2) jvertinimo atlikimas. 

• Konvergentinis pagrjstumas - vienodi dviejg tos pacios sa- 
vybes vertinimg rezultatai, taikant skirtingus metodus. 

• Kritinis multiplizmas - pripazinimas, kad socialiniuose moks- 
luose kiekvienas poziuris dazniausiai turi savo teigiamg ir neigia- 
mg pusig, todel tarn tikro vieno is jg pasirinkimas sqlygoja teorinj 
ar metodinj saliskumq. Kritinis multiplizmas - tai metateorinis mo- 
delis, rodantis, kad naudojami sistemingai suplanuoti daugialy- 
piai poziuriai, siekiant nustatyti saliskumq arba netgi jo isvengti. 


ABSTRACT 

Quality assurance and evaluation research, like other fields of 
social research and its application, are confronted with a series of 
problems. In the present paper, I want first to give a list of such 
problems, although necessarily incomplete. It is then claimed that 
while there is no “perfect” solution to these problems, critical mul- 
tiplism may be a set of approaches which might attenuate the 
problems or at least make them more visible so that one can deal 
with them. Critical multiplism is a general methodological frame- 
work which cannot be fully accounted for here; rather, I want to 
present some major issues and on their base make some sug- 
gestions about how to conduct social research. 


DEFINITIONS OF KEY WORDS 

• Discriminant validity - two assessments of different traits with 
the same or with different methods show different results, thus 
showing that indeed the two traits are distinct. 

• Aptitude - “(A)ny characteristic of a person that forecasts his 
probability of success under a given treatment. (...) Personality 
as well as ability influences response to a given kind of instruction” 
(Cronbach & Snow, 1981 , p. 5). 

• Evaluation- analysis of a practical context with scientific means 
in order to make decisions in this practical context; (1) evaluation 
of institutions: analysis whether the institution achieves its goals 
and how it could be improved; (2) evaluation of programs: analysis 
whether a given program is appropriate to achieve its goals and 
under what conditions it is appropriate to use it. 

• Evaluation research - (1 ) research on evaluation or on evaluation 
studies; (2) doing evaluation. 

• Convergent validity - two assessments of the same trait with 
different methods show the same result. 

• Critical multiplism. Since each approach (in social science) tends 
to have both strengths and weaknesses, the choice of a particular 
one introduces theoretical or methodological biases. Critical 
multiplism is a meta-theoretical framework suggesting that multiple 
approaches are used, yet systematically planned to identify and 
possibly even rule out such biases. 


1 „Kritinio multiplizmo" sqvoka kol kas neturi savo reglamen- 
tuoto lietuvisko atitikmens. „Kritinis multiplizmas" galetg 
buti verciamas kaip ..kritinis daugialypumas", „kritinis jvai- 
rumas“, „kritinio jvairumo metodika" ar pan. Siame straips- 
nyje jis verciamas tiesiogiai. 
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• Pagrjstumas arba tinkamumas - Jsvados teisingumas arba tiks- 
lumas, arba patvirtinimo laipsnis" (Shadish ir kt., 2002, p. 513); 
didelis pagrjstumas rodo nezymiq sisteminp klaidq; pagrjstumui 
patikrinti galima naudoti Campbell ir Fiske (1959, p. 83) sqvok^: 
..Pagrjstumas zymi atitikimqtarp dviejq bandymq pamatuoti t^ pa- 
ciq savybg visiskai skirtingais metodais". 

• Patikimumas arba tiks/umas- tarn tikros charakteristikos verti- 
nimo tikslumas, kad ir kokie butq to vertinimo rezultatai; didelis 
patikimumas rodo nezymiqatsitikting paklaidq; norint patikrinti pa- 
tikimumo lygj, galima pasiremti Campbell ir Fiske (1959, p. 83) 
sqvoka: „Patikimumas-tai atitikimastarp dviejq bandymq pama- 
tuoti tq paci^ savybg visiskai panasiais metodais". 

• Pigmalionas- is graikq mitologijos zinomas Pigmalionas buvo Kip- 
ro karalius, kuris pamilo savo paties sukurtq statul^; Artemide jkvepe 
gyvybp statulai ir Pigmalionas j^vede. Rosenthal ir Jacobson (1968), 
atradg sj efekt^ (destytojai studentg sekmei jkvepia gyvybg), pavadi- 
no jj Pigmaliono efektu (arba Pigmalionu auditorijoje). 

• Saiiskumas - sistemingos klaidos, keliancios pavojq (jvertini- 
mo) pagrjstumui. 


jVADAS 

Maziausiai 80 metLj kalbama, kad socialiniai mokslai isgy- 
vena krizq (pvz., Vygotsky, 1927; Mertens, Fuchs, 1978). 
Kai atliekant tyrimus ir jvertinimus j socialinj elgesj sutel- 
kiamas ypac didelis demesys, susiduriama su daugybe pro- 
blemij. Kai kurios jq isvardijamos zemiau: 

• Priemones, skirtos socialinio elgesio vertinimui, ypac 
klausimynai, yra nelabai patikimi ir nevisai pagrjsti (pig. Mis- 
chel, 1968). 0 tai rodo, kad vertinimai neratikslus ir daznai 
nezinoma, kas is tiesq yra matuojama. Toks dalykas kelia 
daug pavoju bet kuriam tokiu tyrimu pagrjstam sprendi- 
mui (o tai ir yra jvertinimo tikslas), nes pats sprendimo pri- 
emimo pagrindas kelia abejoniq. 

• Atskirq tyrimq rezultatai paprastai neatsikartoja, jeigu 
pakartotinis tyrimas siek tiek skiriasi nuo pirminio origina- 
laustyrimo, netgi irtuo atveju kai skirtumai nezymus; pvz., 
jvertinus sesiq atsitiktinai parinktq pakartotiniq tyrimq re- 
zultatus, beveik 55 proc. is 64 rezultatu parade „duomenq 
pasikartojimo stokq“ (Amir, Sharon, 1991, p. 63); taciau 
pakartotiniq tyrimq kol kas stokojama (ten pat; kt., pvz., 
Karr, Larson, 2005), juos sudetinga paskelbti (Bornstein, 
1991 ; Neuliep, Crandall, 1991), ir tai ypac pasakytina apie 
nesekmingus pakartotinius tyrimus, kuriemstaikomas „nu- 
marinimo stalciuje" principas (Rosenthal, 1979). 

• Tyrinetojai linkq pervertinti savo teorijq pagrjstumo ri- 
bas (Patry, 1991 ; Amir, Sharon, 1991); tai tiesa apskritai pa- 
emus, bet ypac svarbu kai imama kalbeti apie jvertinimus, 
nes jq pagrjstumo ribos nera placios, o tas teorinis pagrin- 
das, kuris siuo atveju padetq, dazniausiai per menkas; visa 
tai taip pat taikytina ir ateities prognozems, kurioms jvertini- 
mo procesuose skiriamas pagrindinis demesys. 

• Jeigu ieskant atsakymo j tq patj klausimq buvo atlikti 
skirtingi tyrimai, tai kuo tikslesni ir labiau tinkami metodai 
taikomi (pvz., kiekybiniai metodai tikslesni uz kokybinius), 
tuo mazesne darosi tikimybe gauti reiksmingq rezultatq. Tai, 
pvz., akivaizdu, lyginant mokslinio simpoziumo metu pateiktu 
skirtingq tyrimq rezultatus; vienqkart esu darps tai neforma- 
liu budu, bet nesu girdejgs apie sistemingus tyrinejimus. 

• Rysys tarp teorijos ir praktikos, kuris laikomas vienu 
svarbiausiq klausimq socialiniuose moksluose apskritai, o 


• Validity- "the truth of, correctness of, or degree of support 
for an inference” (Shadish et al., 2002, p. 513); high validity 
means little systematic error; to check for validity, one can 
use the concept by Campbell and Fiske (1959, p. 83): “Validity 
is represented in the agreement between two attempts to 
measure the same trait through maximally different methods.” 

• Reliability - how precise is the assessment of a particular 
characteristic, whatever it is; high reliability means little random 
error; to check for reliability, one can use the concept by Camp- 
bell and Fiske (1959, p. 83): "Reliability is the agreement 
between two efforts to measure the same trait through 
maximally similar methods”. 

• Pygmalion - in Greek mythology, Pygmalion was a king of 
Cyprus who fell in love with a statue of his own making; Artemis 
gave life to the statue, and he married her. Rosenthal and 
Jacobson (1968) called the effect they found (teachers “make” 
students success come to life) the “Pygmalion effect” (or 
“Pygmalion in the classroom”). 

• Bias - Systematic error jeopardizing validity. 


INTRODUCTION 

At least for 80 years, social sciences have been claimed 
to be in a crisis (e.g., Vygotsky, 1927; Mertens & Fuchs, 
1 978) . Particularly when social behavior is at stake, re- 
search and evaluation encounter many problems. Some 
of them are the following: 

• The instruments to assess social behavior, par- 
ticularly questionnaires, have low reliability and validity 
(cf. Mischel, 1968), which means that the assessments 
cannot be precise and that one does not really know 
what is measured. This jeopardizes particularly any re- 
search-based decision (which is the aim of evaluation) 
because its base is doubtful. 

• Results from particular studies are usually not rep- 
licable if the replication study is slightly different from 
the original one, even if the changes are judged to be 
irrelevant; e.g., in the replications of six randomly cho- 
sen studies, almost 55% of the 64 results indicate “a lack 
of any replicability of data” (Amir & Sharon, 1991 , p. 63); 
however, replication studies are underutilized (op. cit.; 
another example is Karr & Larson, 2005), they are diffi- 
cult to publish (Bornstein, 1991; Neuliep & Crandall, 
1991), and this is even more true for failed replications 
according to the “file drawer effect” (Rosenthal, 1979). 

• Researchers tend to overestimate the domain of 
validity of their theories (Patry, 1991; Amir & Sharon, 
1991); this applies in general but particularly in evalua- 
tions because these are not designed for broad do- 
mains of validity and the theoretical background which 
would help in this regard is mostly insufficient; this holds 
also for predictions for the future which are the main 
focus of evaluations. 

• If different studies are done to answer the same 
question, the more strict and appropriate the methods 
are (e.g., quantitative methods are more strict than quali- 
tative ones), the less likely it is to find significant re- 
sults. This can be seen, e.g., when comparing the re- 
sults of different studies presented within a scientific 
symposium; I did this once informally, but I am not aware 
of a systematic study. 
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ypac jvertinimuose, toli grazu nera taip ispletotas, kad 
praktikq galima butq vertinti kaip teorijos pritaikymq 
(pig., pvz., Bunge, 1967 arba is kitos perspektyvos Ga- 
ge, 1978), isskyrus Lewin (1951, p. 196) tvirtinimq, kad 
nera nieko praktiskiau uz gerqteorijq. 

• Socialiniuose tyrimuose daug artefaktu ir salisku- 
mo (Rosenthal, Rosnow, 1969; Bungard, Bay, 1982; ir 
daugelis kitq) , t.y. savo vertinimuose ir isvadose mes 
darome sistemingas klaidas. 

• |vertinimij procesuose dalyvauja daugybe socia- 
liniu partneriq, kelianciq skirtingus, daznai priestarin- 
gus, reikalavimus (Patry, Gastager, 2004). 

• Tyrinetojai pakliuva j teoriju patvirtinimo ir ju pa- 
neigimo spqstus. Viena vertus, remiantis kritinio ra- 
cionalizmo koncepcija (Popper, 1934/1976; pig. ze- 
miau) tikslas yra paneigti teorijas. Kita vertus, „atski- 
riems mokslininkams, kaip ir visiems zmonems, ge- 
riau pavyksta pastebeti savo teorijij patvirtinimq, o 
ne paneigimq (Faust, 1984; Mahoney, 1976)“ (Sha- 
dish, 1986, p. 79). Paprastai tyrimo tikslas nera pa- 
neigti kazkieno teorijq, kuria remtasi; dazniau nusivi- 
liama tada, kai nepavyksta jrodyti, jog musq hipote- 
zes pasitvirtino. 

Paminetina ir moksliniq tyrimu apskritai, o ypac so- 
cialiniu mokslu tyrimq kritika metateoriniame lygmenyje, 
t.y., mokslofilosofijos arbafilosofinesepistemologijos lyg- 
menyje. Cia isvardijama keletas kritikuojamq dalykq: 

• Savo moksliniq revoliucijq koncepcijoje Kuhn 
(1962) pabrezia, kad mokslas nesipletoja tolygiai, bet 
patiria luzius ir revoliucijas, taciau socialiniai mokslai 
nera pasiekq net sio etapo, greiciau tik ikimokslini. 

• Feyerabend (1975) nuomone, ideja, kad mokslas 
gali ir turetq buti pletojamas pagal nekintamas ir uni- 
versalias taisykles, per daug naivi ir zalinga mokslui. 
Visos metodikos turi savo trukumu. Jo nuomone, 
„mokslas yra is esmes anarchistinis sumanymas: teori- 
nis anarchizmas yra humaniskesnis ir tinkamesnis pa- 
zangai skatinti negu jo jstatymq ir tvarkos alternatyvos" 
(ten pat, p.1 7), tad jis kelia klausimq, ar turime „is tikrq- 
jq patiketi, kad naivios ir kvailokos taisykles, kuriomis 
vadovaujasi mokslininkai, gali isaiskinti tokj „sqveikq la- 
birintq" (p. 17 ir t.t.), kur pastaroji sqvoka apima „zmo- 
gaus kaitos sudetingumq ir nenuspejamq bet kokio vei- 
kejo ar zmogaus sprendimo galutiniu padariniq pobu- 
di“ (ten pat, p. 17). 

• Glasersfeld (1 995) teigia, kad nera jokio budo su- 
zinoti tiesq arba „kas yra is tikrqju“. Greiciausiai, vis- 
kas, kqzinome, yra musq interpretacija (musq isgalvo- 
jimas, sako Foerster, 1998). Netiesa, o pasiekiamumas 
turetq buti demesio centre, be to, yra jvairiq pasiekia- 
mumo kriterijq (Patry, 2001a). 

• Remiantis sistemq teorija (pvz., Watzlawik ir kt., 
1974) galima manyti, kad viskas susij§ su viskuo; ne- 
galima izoliuoti kokios nors vienos sistemos ir jsivaiz- 
duoti, kad nustateme, kaip ji is tikrqjq veikia. Kiti auto- 
riai yra teigp, kad tikrove pernelyg sudetinga, kad ji bu- 
tq suprasta, naudojantis mokslinemis (kiekybinemis) 
priemonemis (pvz., Cziko, 1989). 

• Popper (1934/1976) pabreze, kad indukcinis me- 
todas yra nejmanomas. Taciau jo pasiulytas sprendi- 
mas - paneigimas, o ne patvirtinimas - yra nejmano- 
mas, kai teiginiai yra tikimybiniai, o tokie teiginiai dazni 
socialiniuose moksluose. 


• The relationship between theory and practice, one of 
the most important issues of social science in general and 
of evaluation in particular, is far from being developed in 
such a way that practice can be seen as application of 
theory (cf., e.g., Bunge, 1967, or, from another perspec- 
tive, Gage, 1978), despite Lewin’s (1951 , p. 196) claim that 
there is nothing as practical has a good theory. 

• There are many artefacts or biases in social research 
(Rosenthal & Rosnow, 1969; Bungard & Bay, 1982; and 
may more), i.e., we make systematic errors in our assess- 
ments and inferences. 

• In evaluations, there are many stakeholders with many 
different requirements, some of which are contradictory 
(Patry & Gastager, 2004). 

• Researchers are caught in a confirmation-refutation 
trap. On one hand, according to the concept of critical ra- 
tionalism (Popper, 1934/1976; cf. below), their aim is to re- 
fute theories. On the other hand, “individual scientists, just 
like all human beings, are better at noting confirmations of 
their theories than disconfirmations (Faust, 1984; Mahoney, 
1976)” (Shadish, 1986, p. 79). Typically, the aim of doing a 
study is not to refute one’s theory which usually is some- 
thing one believes in; rather, we are disappointed if we do 
not succeed to show that our hypotheses are confirmed. 

In addition, there have been several criticisms of scien- 
tific research in general and of social science research in 
particular on a meta-theoretical level, i.e., from the stand- 
point of philosophy of science or philosophical epistemol- 
ogy. Some of these criticisms are: 

• With his concept of scientific revolutions Kuhn (1 962) 
emphasizes that science is not growing steadily, but that 
there are ruptures and revolutions - however, social sci- 
ences are not even in that stage, but rather “pre-scientific”. 

• Feyerabend (1975) argues that the idea that science can, 
and should, be run according to fixed and universal rules is 
both too simple and detrimental to science. All methodologies 
have their limitations. For him, “science is an essentially anar- 
chistic enterprise: theoretical anarchism is more humanitarian 
and more likely to encourage progress than its law-and-order 
alternatives” (op. cit., p. 17), and he asks whether we are “re- 
ally to believe that the najve and simple-minded rules which 
methodologists take as their guide are capable of accounting 
for such a ‘maze of interactions’” (op. cit., pp. 17f.), the latter 
term referring to the “complexity of human change and the 
unpredictable character of the ultimate consequences of any 
given actor or decision of men” (op. cit., p. 17). 

• Glasersfeld (1995) criticizes that there is no way we 
can know truth or “what is the case”. Rather, all we can 
know is our own construction (our invention, says Foerster, 
1998). Instead of truth, viability should be at stake, and 
there are different viability criterions (Patry, 2001a). 

• Based on systems theory (e.g., Watzlawik et al., 1 974) 
one can assume that everything is connected with every- 
thing; one cannot isolate one system and pretend to have 
identified how it really works. Others have said that reality 
is too complex to catch it with scientific (quantitative) means 
(e.g., Cziko, 1989). 

• Popper (1934/1976) has pointed to the impossibility 
of induction. His proposed solution - refutation instead of 
verification -, however, is impossible if the statements are 
probabilistic, which is usually the case in social sciences. 

• Theoretical statements are abstract. The more con- 
crete they are, the smaller are the respective domains of 
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• Teoriniai teiginiai yra abstraktus. Kuo jie konkretesni, 
tuo siauresnes atitinkamos pagrjstumo ribos; kuo plates- 
nes teiginio pagrjstumo ribos, tuo jis abstraktesnis (Her- 
rmann, 1979, p. 160-161). Tai abstraktumo ir konkretumo 
dilema. 

• Teorijos paprastai negali gerai paaiskinti socialinio 
elgesio situacijos specifiskumo, o tai kelia daugybq pro- 
blemq (Patry, 2001b). 

Galima butq isvardinti ir dar daugiau kritiniq pastabq 
del socialiniq mokslu ir ju tyrimq. Atrodo, tarsi tyrimus is 
visij pusiq suptq pavojai, tad kyla klausimas, ar socialiniai 
mokslai kada nors gales duoti naudingu rezultatq. Is tikrq- 
ju palyginus su kitais mokslais, pvz., fizika, chemija ar net- 
gi biologija, matyti, kad socialiniai mokslai ne tokie tikslus 
(pig., pvz., Hedges, 1987) ir maziau apibrezti. 

Sios problemos daro jtakq ir jvertinimo tyrimams bei 
kokybes uztikrinimui. Visos auksciau issakytos kritines pa- 
stabos gali tikti ir, pavyzdziui, programq vertinimams sie- 
kiant patobulinti destymo kokybq. Geras bandymas, ku- 
riuo cia pasiremti negalime, butq pasirinkti bet kurj tyrimq 
ir jj isanalizuoti remiantis auksciau pateiktomis kritinemis 
pastabomis. Daugeliu atvejq tik vienas kitas rezultatas lik- 
tu galioti. 

Socialiniuose tyrimuose egzistuoja keturi budai cia ap- 
tartoms problemoms sprqsti; (a), (b) ir (d) paimta is Bun- 
gard ir Bay (1982), (c) sqlygoja bendroji patirtis: 

(a) Problemas galima ignoruoti ir manyti, kad sociali- 
niai mokslai neisgyvena jokios krizes. Tai dazniausia reak- 
cija, kuri psichologiskai suprantama, taciau problemq ne- 
igimas niekada nebuvo teisingas sprendimas, jis negali pa- 
deti nei mokslui zengti j priekj, nei issprqsti egzistuojanciq 
problemq. 

(b) Problemos galima pameginti isvengti renkantis ta- 
riamai nepriekaistingqmetodologinq nuostatq. Pavyzdziui, 
siekdami socialinio pobudzio tyrime isvengti artefakto pro- 
blemos, kylancios del subjekto reaktyvumo (zinodami, kad 
yra stebimi, subjektai elgiasi kitaip), tyrinetojai pasitelkia 
reaktyvumo nesukeliancius metodus (Webb ir kt. , 1981). 
Toks pasirinkimas galetu padeti issprgsti vienq problemq, 
taciau kitos neisnyksta, o joms gali buti taikomi tie patys 
argumentai kaip ir (a) atveju. 

(c) Moksliniu metodu priimtinumq galima apskritai pa- 
neigti ir rinktis kitus poziurius, pvz., ezoterizmq. Taciau tai 
nera geriausias budas, nes susidurimas su problemomis ne- 
ra priezastis atsisakyti tyrimq ir griebtis kitos tariamai malo- 
nesnes veiklos. Moksliniai tyrimai gali tureti problemq; ta- 
ciau kad ir kokie butq ju trukumai, is viso to kq turime jie 
neabejotinai liekageriausi irtinkamiausi pazinimo budai pa- 
lyginti su visais kitais budais, kuriais siuloma tyrimus pakeisti. 

(d) Problemas galima meginti isaiskinti teoriniame ir me- 
todiniame lygmenyje, ir jeigu jos neissprendziamos, tai bent 
jau suvokiamos ir pritaikomi principal, kurie padeda (i) nu- 
statyti, ar problema egzistuoja ir (ii) apriboti jos poveikj. 
Kaip matysime veliau, kritinis multiplizmas siuo atveju gali 
pasiulyti potencialius budus tyrimus atlikti geriau negu jie 
buvo atliekami iki siol. 

Is auksciau pateiktu pasiulymq (d), be abejones, yra 
priimtiniausias. Galima suformuluoti tokiq pagrindinq tai- 
syklq: 

1 pasiulymas: Pripazinti social in iy mokslu tyrimij prob- 
lemas, bet interpretuoti jas kaip issukius ir kiek jmanoma 
jas sumazinti! 


validity; the broader the domain of validity of a statement 
is, the more abstracts is (Herrmann, 1979, pp. 160-161): 
This is the generalizability-concreteness dilemma. 

• Theories usually do not account sufficiently for 
situation specificity of social behavior, which creates 
many problems (Patry, 2001b). 

One could mention many more criticisms to social 
science and research. It seems that research is jeopar- 
dized from many sides, and one wonders whether so- 
cial science will ever be able to bring about useful re- 
sults. Indeed, when compared with other scientific dis- 
ciplines such as physics or chemistry or even biology, 
one must say that the social sciences are much softer 
(cf. , e.g., Hedges, 1987) and fuzzy. 

These problems have their impact also on evalua- 
tion research and quality assurance. All of the above 
criticisms can be applied, for instance, to assessments 
within programs to improve quality of teaching. It would 
be a good exercise - which cannot be done here - to 
take any piece of research and to check whether it holds 
against the criticisms presented above. In most cases, 
very little of the results will survive. 

In social research, there have been four ways to deal 
with these kinds of problems; (a), (b), and (d) are from 
Bungard and Bay (1982), (c) is a common experience: 

(a) One can just ignore the problems and presume 
that there is no crisis in social research at all. This is 
the most frequent reaction. This reaction is psycho- 
logically understandable, yet denial of problems is 
never a good reaction, and it will certainly contribute 
neither to the progress of the science nor to solving 
the problems. 

(b) One can try to avoid problems by using presum- 
ably flawless approaches. For instance, to avoid the 
artifact problem in social research due to the subjects’ 
reactivity (the subjects’ knowing that they are being ob- 
served makes them behave differently), researchers use 
nonreactive methods (Webb et al., 1981). Such a reac- 
tion might help to solve one problem, the others, how- 
ever, are not dealt with, and for them, the same argu- 
ment holds as for (a). 

(c) One can deny the appropriateness of scientific 
methods at all and to turn to other approaches such as 
esotericism. This is not appropriate since the existence 
of problems is not a reason to give up research and to 
turn to presumably more gratifying activities. Research 
may have its problems; however, as bad as it may be 
yet, it is still the best that is available and definitely more 
appropriate than all approaches which have been pro- 
posed as substitutes. 

(d) One can try to account for the problems on theo- 
retical and methodological levels and if not solve the 
problems at least be aware of them and apply principles 
that permit (i) to identify whether there are problems and 
(ii) to minimize their impact. As will be discussed below, 
critical multiplism can be an approach in this sense with 
the potential to do better than research until now. 

From the suggestions above, (d) is certainly the most 
appropriate. As a general rule, we can formulate the 
following 

suggestion 1: Acknowledge the problems of social 
research, but interpret them as challenges and do your 
best to reduce them to a minimum! 
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1 

KRITINIS MULTIPLIZMAS 

1.1. KRITINIO MULTIPLIZMO PRINCIPAI 
IR LOGINIS PAGRINDAS 

Kiekvienas poziuris - vertinimo priemone, metodas, te- 
orija, kt. - yra saliskas, turi savo „silpnq vietq" arba „tam- 
siu demiu“, is kuriq keletas pamineta jvade pateiktame 
problemu sqrase. Kai vadovaujamasi tik vienu poziuriu, 
saliskumas uzgoziamas ir jj sunku pastebeti. Butent cia 
kritinio multiplizmo koncepcijos taikymas gali padeti. 

Trumpai tariant, kritinis multiplizmas reiskia, kad nau- 
dojami daugialypiai poziuriai atsakyti j tam tikrq (tyri- 
mo) klausimq. Taciau visai nereikalingas neprotingas 
aktyvumas, kai „bet kas tinka, svarbu padaryti kiek jma- 
noma daugiau". Kritinis multiplizmas yra kritinis siste- 
minis budas, leidziantis apimti kuo daugiau jvade is- 
keltq problemu, kad jq isvengti (Shadish, 1994). Het- 
herington (1997) pasiule, kaip atskirti kritinj multipiliz- 
mq nuo kitu budq (1 lentele); jis pasireme „apmqstyta- 
neapmqstyta" skirtumu analizuodamas metodus, bet tai 
gali buti taikoma ir kitoms sritims, pavyzdziui, teorijoms. 

Kritinis multiplizmas laikosi nuostatos, kad nera ne 
vieno poziurio, kuris butq laikomas geresniu uz kitus; 


1 

CRITICAL MULTIPLISM 

1 .1 . PRINCIPLES OF CRITICAL MULTIPLISM AND ITS 
RATIONAL 

Every single approach - assessment tool, method, theory, 
etc. - has its biases, its “black spots”, some of which are 
pointed at in the list of problems mentioned in the first sec- 
tion. Using only a single approach will obscure these bi- 
ases so that they cannot be recognized. This is where criti- 
cal multiplism can provide a remedy. 

Briefly stated, critical multiplism means that to use mul- 
tiple approaches to answer a particular (research) ques- 
tion, yet not in the sense of a mindless activism of the type 
“anything goes, just do as much as possible”, but rather in 
a critical and systematic way so that as many problems 
among those mentioned in section 1 and others are ac- 
counted for and - possibly - ruled out (Shadish, 1994). 
Hetherington (1997) proposed the systematic as in table 1 
to distinguish critical multiplism from other approaches; he 
applied this distinction to methods, but it can be extended 
to other domains, like theories. 

Critical multiplism means that there is not one single 
best approach; rather, each method has its advantages and 


1 lentele. Kritinio multiplicizmo ir kiti) poziuriq skirtumai (pagal Hetherington. 1997, 18 skaidre) 
Table 1 : Distinction of critical multiplism from other approaches (from Hetherington, 1 997, slide 1 8) 



NEAPMAjSTYTA 

APMASTYTA 


THOUGHTLESS 

THOUGHTFUL 

PAVIENIS METODAS 

Beprasmis monizmas 

Tikslus monizmas 


Skurdi teorija 

Niutono mokslas 


Zemas pagrjstumas 

MAX-MIN-CON 2 

SINGLE METHOD 

Mindless Monism 

Rigid Monism 


Poor science 

Newtonian Science 


Rear-End Validity 

MAX-MIN-CON 2 

DAUGIALYPIAI METODAI 

Beprasmis daugialypumas 



Skurdi teorija 
Bet kas tinkama 

Kritinis multiplizmas 

MULTIPLE METHODS 

Mindless Multiplism 



Poor science 
Anything Goes 

Critical Multiplism 


2 MAXMINCON principq suformulavo Kerlingeris. Tai yra klasikinis 
eksperimentinio tyrimo principas. Jis reiskia didink kiek jma- 
noma sistemq skirtingumq. Mazink kiek jmanoma paklaidq 
skirtumq. Kontroliuokklaidinanciuskintamuosius. Idejayrata, 
kad siekiama dideliq skirtumq tarp eksperimentines ir kontro- 
linesgrupiq kaip parenkamq sistemq (t.y., procedures turi bu- 
ti veiksmingos), mazi skirtumai grupiq viduje (homogenines 
grupes, auksto patikimumo ir tikslumo vertinimo instrumen- 
tal) ir kad kintamieji, kurie gali sukelti zenklias klaidas turi buti 
kontroliuojami (t.y. bet koks veiksnys, kuris gali daryti papildo- 
mq jtakq, turi buti eliminuotas). Tai budinga laboratoriniams 
eksperimentams, kuriais galima aiskiai parodyti, kad vienas 
specifinis nepriklausomas kintamasis (kokia nors veikla) turi 
itakos kokiam nors priklausomam kintamajam (kuris yra ma- 
tuojamas). Realiame gyvenime vis del to negalima eliminuoti 
visas itakas, taigi maxmincon teigia, kad eksperimentai stoko- 
ja (ekologinio) pagrjstumo, t.y. tokiq eksperimentq isvados ar 
teiginiai neatitinka to, kas vyksta realiame gyvenime. 


2 MAXMINCON principle stems from Kerlinger and is classiscal 
principal from experimental research; it means Maximize 
systematic variance. Minimize error variance. Control for 
confounding variables. The idea is that one should look to 
get high differences between experimental and control 
groups (i.e. , powerful treatments), little variance within the 
groups (homogeneous groups, highly reliable assessment 
instruments), and that variables which might produce sys- 
tematic errors should be controlled (i.e., any factor that 
might have additional influences should be eliminated). This 
yelds laboratory experiments in which one can powerfully 
demonstrante that one specific independent variable (treat- 
ment) has an influence on one specific dependet variable 
(which is measured). However, life is not such that one can 
eliminate other influences etc., so the maxmincon - experi- 
ments lack ecological validity, i.e. the statements from such 
experiments do not permit statements about what is hap- 
pening in real life. 
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manoma, jog kiekvienas metodas turi savo pranasumq ir 
trukumu. Todel siuloma taikyti keletq metodq vienu metu. 
Jeigu taikant skirtingus metodus gaunami panasus rezul- 
tatai, galimatiketi jq pagrjstumu, kol nera priesingq jrody- 
mq. Jeigu rezultatai labai skiriasi, akivaizdu, kad yra pro- 
blemq: zinome, kad rezultatais pasitiketi negalime, bet ne- 
zinome, kurie is jq patikimesni lyginant su kitais, ir kuriuo 
atzvilgiu jie laikytini patikimesniais. 

Zemiau pateikiami kritinio multiplizmo pavyzdziai (taip 
patzr. Houtsirkt., 1986; Patry, 1989; Shadish, 1 986 ir kt.) : 

• Daugialypis operacionalizmas ir daugialypiai meto- 
dai (Campbell, Fiske, 1959); 

• Sudetiniq daugiamaciq modeliq analizes taikymas vie- 
toj vienamaciq modeliq (tai susijg su konstruktais, kurie bus 
tiriami) ; 

• Daugialypiq konkuruojanciq hipoteziq patikrinimas 
(Campbell, 1969); 

• Daugialypiq susijusiqtyrimq taikymas, pavyzdziui, ty- 
rimq programose (Herrmann, 1976), tyrimq strategijose, ap- 
imanciose keletq skirtingai planuotq vertinimu tokiq, kaip 
pvz., ivairiapusiai tyrimai (Seashore, Louis, 1984; Herriott, 
Firestone, 1983; Slavin, Madden, 1993; Audet, d’Amboise, 
2001) ir kt. 

• Skirtingq nepriklausomai atliktu tyrimq rezultatq sin- 
teze (pvz., metaanaliziu budu; Cook, Leviton, 1980; Cook, 
1991 ; Glass, McGaw, Smith, 1981 ; Rosenthal, 1989; irt.t.); 

• Tq paciq duomenu nagrinejimas taikant skirtingus sta- 
tistinius metodus; 

• Skirtingq teorijq taikymas hipotetiniams atsakymams 
j panasius tyrimq klausimus gauti ir rezultatams interpre- 
tuoti (teorijq konkurencija, koegzistencija ir netgi jq papil- 
domumas, zr., pvz., Patry, 1998 ). 

Kaip mineta auksciau (1 lentele), kritinis multiplizmas 
turi buti atskirtas nuo betikslio multiplizmo. Shadish (1 986) 
pritaria minciai, kad jvertintojai „taiko nepaprastai platq me- 
todq ir savarankiskq teorijq spektrq" (p. 75). Taciau sis mul- 
tiplizmas renkantis „teorijas ir metodus" nera susijgs su jo- 
kia sqmoningai susiformavusia teorija, kurioje butq teigia- 
ma, kad mokslq geriausiai pletoti butent tokiu budu. Grei- 
ciausiai tai buvo sukelta vidiniq ir isoriniq tarn tikros profe- 
sijos veiksniu" (p. 76). Toks daugialypumas yra atsitiktinis, 
t.y. betikslis uzuot buvgs sistemingas arba, kaip sakoma 
koncepcijos pavadinime, „kritinis“. 

Priestaraudami sitokiam betiksliam multiplizmui, Wil- 
liams Shadish, Thomas Cook ir sio straipsnio autorius is- 
kelia minti, jog siekiant efektyviai atlikti tyrimq ir jvertinimq 
bOtina taikyti multiplizmo sistemingai nepaisant kritiniq pa- 
stabu, issakytu jvade. Suplanuoto multiplizmo esme, re- 
miantis Shadish (1986, p.78), „nereiskia, kad kuodaugiau, 
tuo geriau: daugiau veiksmq, daugiau metodq, daugiau 
priemoniq, daugiau teorijq. Daug svarbiau mokslininkus ap- 
rupinti priemonemis ir padeti jiems tirti savo pazinimo ri- 
bas." 

Tad „vienas svarbiausiq pozymiu yra heterogenisku- 
mo nurodymas; svarbus ne taikytq pasirinkimq skaicius, 
bet labiau tai, kad butq pasitelkti jvairus pasirinkimai, ku- 
riu saliskos nuostatos veikia skirtingomis kryptimis" . Dau- 
gialypumas tikslingas tuo atveju, jeigu ir tik jeigu suteikia 
ivairumo toms problemoms, kurios buvo aptartos jvade ir 
leidzia atskleisti saliskumo, jau minetq sio skyriaus pra- 
dzioje. 

Del sios priezasties siekiant gebejimo taikyti sistemini 
(arba kritinj) multiplizmo, svarbu issamiai suprasti mokslo 


its problems. It is suggested, then, to use several ap- 
proaches simultaneously. If the different approaches 
yield similar results, we can trust that there is some- 
thing reasonable in them, until there is evidence of 
the opposite. If they differ considerably, it is obvious 
that there is a problem: We know that we cannot trust 
the results, but we do not know which ones are more 
trustworthy and which ones are less, or in what re- 
gard they are trustworthy. 

The following are examples of critical multiplism 
(see also Houts et al., 1986; Patry, 1989; Shadish, 
1986; etc.): 

• Multiple operationalism and multiple methods 
(Campbell & Fiske, 1959); 

• Analysis of complex multivariate instead of 
univariate models (this refers to the constructs to be 
analyzed); 

• Checking multiple rival hypotheses (Campbell, 
1969); 

• Using multiple related studies, as in research pro- 
grams (Herrmann, 1976), research strategies with sev- 
eral differently designed assessments such as multi- 
site studies (Seashore Louis, 1 984; Herriott & Firestone, 
1983; Slavin & Madden, 1993; Audet & d'Amboise, 
2001), etc. 

• Synthesis of results of independent studies (e.g., 
through meta-analyses, Cook & Leviton, 1980; Cook, 
1991 ; Glass, McGaw & Smith, 1981 ; Rosenthal, 1989; 
etc.); 

• Analysis of the same data with different statisti- 
cal approaches; 

• Use of different theories to give hypothetical an- 
swers to similar research questions and to interpret re- 
sults (competition, coexistence or even complemen- 
tarity of theories, see, e.g., Patry, 1998 and below). 

As said above (see also table 1), critical multiplism 
is to be distinguished from mindless multiplism. 
Shadish (1986) sustains that evaluators “have used 
an unusually wide array of methods and substantive 
theories” (p. 75). However, this “multiplism in theory 
and method choice mostly has not been due to any 
conscious theory that science is best conducted in 
such a fashion. Rather, it probably was elicited by in- 
ternal and external sociological forces in the profes- 
sion.” (p.76) This kind of multiplism, hence, is hapha- 
zard, i.e., mindless, instead of being systematic or, as 
the name of the approach suggests, “critical”. 

In contrast to this mindless multiplism, authors like 
William Shadish, Thomas Cook and the present author 
suggest that in order to achieve the goal of doing valid 
research and evaluation despite the criticisms mentioned 
above, it is necessary to apply multiplism systematically. 
The point of planned multiplism, according to Shadish 
(1986, p. 78), “is not that more is better - more opera- 
tions, more methods, more measures, and more theo- 
ries. Rather, the point is to provide tools to help scien- 
tists explore the boundaries of their knowledge.” 

And for this, “one of (the) key features is a prescrip- 
tion for heterogeneity; it is not the sheer number of im- 
plemented options that are important, but rather that 
the different options be implemented that have biases 
operating in different direction^’ (italics added). Multi- 
plism achieves its goals if and only if it provides diver- 
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kritikq ir meginti paaiskinti jq remiantis jvairiais poziu- 
riais. Tad cia svarbus: 

2 pasiulymas. Suvokti ga/imq savo tyrimo satisku- 
mq ir parengti atitinkamas gynybos strategijas pasitei- 
kiant daugialypius modeiius! 


2 

PARADIGMOS IR TEORIjy 
PLIURALIZMAS 

Bendrasis metodologijos principas, kuris taip pat gali buti 
pritaikytas jvertinimams yra tas, kad net geriausias tyrimo 
sumanymas ir vertinimo priemones negali apsaugoti tyri- 
mo rezultatu, jeigu tas tyrimas grindziamas netinkama teo- 
rija. Todel pirmiausiai noreciau aptarti paciq teorijq, kuria 
tyrimas grindziamas, o po to kalbeti apie vertinimus. Ne- 
paisant visos ju svarbos, teorijos klausimams buvo skiria- 
ma daug maziau demesio nei vertinimo klausimams. Ta- 
ciau kritiniam multiplizmui teoriju pliuralizmas yra pats svar- 
biausias klausimas. Galima pridurti, jog sis klausimas svar- 
besnis jvertinimo tyrimuose, nes teorinis pagrindas siuose 
tyrimuose yra labai jautri tema. 

2.1 . TEORIJOS (ARBA TEORIjy) VAIDMUO 
jVERTINIME 

Manoma, kad daugelyje jvertinimo tyrimij teorijos nera 
svarbios. Is tiesu tiriamas klausimas jvertinimo studijoje 
yra ne tai, ar pati teorija teisinga, ar ne (kad ir kq reikstq 
pasakymas „buti teisingam", nes, pavyzdziui, remiantis 
konstruktyvistq epistemologija, isvis nera budq suzinoti 
tiesq, todel Glasersfeld (1981) kalba ne apie tiesq, o 
apie „pasiekiamumq“ - siame straipsnyje sis klausimas 
smulkiau nenagrinejamas). Daug svarbiau, ar konkreti 
programa veiksminga (pvz., Patry, Perrez, 2000), ar kon- 
kreti institucija pasiekia savo tikslus (pvz., Thonhauser, 
Patry, 1999); is pirmo zvilgsnio atrodo, kad siems tiks- 
lams teorija nera butina. 

Taciau reikia priminti, kad mokslas (jskaitant ir iverti- 
nimq) prasideda nuo klausimij ir problemij kelimo, po jq 
priimami preliminarus sprendimai (Popper, 2002). Taciau 
sie preliminarus sprendimai butinai yra teoriniai, nesvar- 
bu, ar tai pripazjstama, ar ne. Kalbant toliau, butina pri- 
durti, kad bet kuris stebejimas yra susijqs su tarn tikra 
teorija (tai taip pat akcentavo Popper, 2002). Tai taip pat 
budinga ir konstruktyvistq epistemologineje tradicijoje, 
Glasersfeld, 1981). Tie, kurie jsivaizduoja, jog nesiremia 
jokia teorija, vis delto taiko subjektyvias teorijas arba ne- 
profesionalias teorijas (Furnham, 1998) su visomis is ju 
kylanciomis problemomis: jos yra nesistemiskos, del „pa- 
tvirtinimo-paneigimo" spqstu (zr. auksciau) dar saliskes- 
nes, pats jq turinys atsitiktinis ir t.t., - bet vis vien tai tarn 
tikro pobudzio teorijos. Taigi, bet kuriuo atveju geriau 
vadovautis aiskiomis teorijomis, t.y. remtis moksliniais 
tyrimais, nes bet kuriame socialiniame tyrime vis dar likq 
daug „pasleptq“ teorijq. 

3 pasiulymas: prisipazinkite, kad jus taikote teorijas 
ir aiskiai jas isdestykite! 

2.2. AISKINIMy DAUGIALYPUMAS 

Reiskiniai paprastai turi ne vienq priezastj; todel cia 
tiktq taip vadinamas „INUS“ modelis (angi a cause is 
“an insufficierttbuX non-redundant<parX of an unneces- 
sary but sufficient condition" - priezastis yra „nepa- 


sity with respect to the problems addressed in the introduc- 
tion and hence permits to uncover the biases mentioned at 
the beginning of the present section. 

For this very reason, in order to be able to apply system- 
atic (or critical) multiplism, it is essential to be aware of the 
criticisms to science in detail and to try to account for them 
through varying approaches. Hence the 

suggestion 2: Know the potential biases of your re- 
search and work out appropriate defense strategies through 
multiple approaches! 

2 

PARADIGMS AND PLURALISM 
OF THEORIES 

A general principle in methodology that holds also for evalu- 
ations is that the best research design and assessment 
tools cannot save a study with a bad theory. Therefore, I 
want first to look at the theory underlying a study, and then 
go on with assessments. Despite its importance, issues of 
theory have much less been looked at than issues of as- 
sessment. In critical multiplism, the pluralism of theories is 
at stake. One might add that this question is the more im- 
portant in evaluation research because the theoretical back- 
ground in evaluation studies is a very sensitive topic. 

2.1 THE ROLE OF THEORY (OR THEORIES) 

IN EVALUATION 

In many evaluation studies, theories supposedly are not rel- 
evant. Indeed, the research question in an evaluation study 
is not whether a theory is true or not (whatever “being true” 
might mean, since for instance according to constructivist 
epistemology, there is no means at all to find out truth, and 
Glasersfeld, 1981, consequently speaks of “viability” instead 
of truth - it is not the place here to go into details in this 
regard). Instead, the question is whether a specific program 
works (e.g., Patry & Perrez, 2000) or a specific institution 
achieves its aims (e.g., Thonhauser & Patry, 1999); at first 
sight, for these aims, a theory is not necessary. 

However, it is to be reminded that science (including 
evaluation) starts with questions and problems, and is fol- 
lowed by tentative solutions (Popper, 2002). These tenta- 
tive solution, however, are necessarily theoretical, whether 
acknowledged or not. One must add, further, that any ob- 
servation is also theory-bound (this has also been stressed 
by Popper, 2002, but also in the constructivist epistemo- 
logical tradition, Glasersfeld, 1981). Those who pretend not 
to use any theory actually use subjective theories or lay 
theories (Furnham, 1988) with all its problems: They are 
unsystematic, the confirmation-refutation trap (see above) 
is even more biasing, the content is haphazard, etc. - yet it 
is still some kind of theory. So in any case it is better to 
have explicit theories and so to capitalize on scientific re- 
search - there is still much implicit theory in any social re- 
search left. Hence the 

suggestion 3: Acknowledge that you use theories and 
make them explicit!. 

2.2. MULTIPLE EXPLANATIONS 

Phenomena usually do not have only one cause; rather, 
the “INUS” concept is appropriate (a cause is “an insuffi- 
cient but non-redundant part of an unnecessarybuX suffi- 
cient condition; Mackie, 1974, p. 62, italics in the origi- 
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(d) 

PAKANKAMA SALYGA („S“) 
SUFFICIENT CONDITION (“S”) 

(a) 

KITOS SALYGOS („l“) 
OTHER CONDITIONS (“I”) 


(b) 

1 PRIEZASTIS („N“) 
CAUSE 1 (“N”) 


EFEKTAS 

EFFECT 


(C) 

KITOS PAKANKAMOS SALYGOS (U 1 PRIEZASCIAI) 
OTHER POSSIBLE CONDITIONS (U FOR CAUSE 1) 

2 PRIEZASTIS /CAUSE 2 


1 pav. INUS salygos (zr. teksta) 
Figure 1 : INUS-conditions (see text) 


kankama, taciau ne pertekline dalis nebOtinos, bet pa- 
kankamos salygos" (Mackie, 1974, p.62). 3 INUS prin- 
cipal galetq buti pavaizduoti sitaip: (1 pav.; kaip pavyz- 
dys naudojamas klasikinis saliskumo socialiniuose ty- 
rimuose atvejis: eksperimentatoriaus laukimo efektas, 
Rosenthal, 1976 pavyzdyje „Pigmalionas auditorijoje", 
Rosenthal, Jacobson, 1968): 

(a) /. Tai (potenciali) priezastis, bet pati savaime ji 
nepakankama, ypac veiksminga kaip priezastis ji gali tap- 
ti tiktai tada, kai sudaroma grupe kitq sqlygq (1 pav. „ki- 
tos sqlygos"). Pigmaliono atveju destytojai yra jsitikinp, 
jog kai kurie jq studentai pasizymi ypatingais gabumais, 
padedanciais jiems „prazysti“ („atskleisti savo gabu- 
mus"), t.y. sparciai tobuleti. Savaime sie gabumai yra 
nepakankami, nes be destytojo darbo jie nebus veiks- 
mingi. 

(b) N. Si (potenciali) priezastis nera pertekline viso 
veiksniq „rinkinio“ dalis; tai reiskia, kad „rinkinys“ nebus 
veiksmingastol, kol (potenciali) priezastis netaps butina 
jo dalimi. „Pigmaliono“ atveju „rinkinys“ apima auksciau 
minetus gabumus ir tinkamq mokymq. Destytojai mano, 
jog ugdyti sekmingiau sekasi „greitai prazystancius" („at- 
siskleidziancius"), negu kitus studentus. 

(c) U. Sis „rinkinys“ yra nebOtinas, galetq buti ir kitq 
panasaus poveikio „rinkiniu“, kai nesama arba esama 
pirmojo „rinkinio“ ar jo elementq, bet nera (potencialios) 
priezasties. „Pigmaliono“ atveju galima butu diskutuoti 
su Rosenthal ir Jacobson, kad ne studentu gabumai pa- 
spartinajqtobulejimq, bet labiau destytojutikejimas, ver- 
ciantis juos kitaip elgtis su potencialiai greitai „prazys- 
tanciais („atsiskleidzianciais“) studentais. 

(d) S. Sis „rinkinys“ yra pakankamas, jeigu turimas 
visas toks „rinkinys“, rezultatas greiciausiai bus pasiek- 
tas, esant bet kokioms sqlygoms. „Pigmaliono“ atveju 
destytojai mano, jog butent „prazydimo“ (atsiskleidimo) 
potencialas, bet ne jq tikejimas ir su tuo susijgs elgesys 
yra spartejancio tobulejimo priezastis. 

Is Pigmaliono pavyzdzio matyti, kad taikant „INUS“ 
salygos modelj, nagrinejamas gali buti ir saliskumas, 
pvz., kaip 2 priezastis (1 pav., c komponentas) ir pasiu- 
lyta alternatyvi laukiamo rezultato interpretacija. 

3 Sis teiginys gali buti suformuluotas ir pozityvia nuostata. Tada jis 
skambetq taip: ..Priezastis yra esmine s^lygq visumos dalis, 
kurios neisvengiamai veda link pasekmes, bet si pasekme taip 
pat gali tureti ir kitas priezastis, negu tik tq, kuria tyrinejame". 


nal). 3 The INUS principles can be described as follows 
(see also figure 1 ; as example, I use a classical bias in 
social research: the experimenter expectancy effect, 
Rosenthal, 1976, in the example of “Pygmalion in the 
classroom”, Rosenthal & Jacobson, 1968): 

(a) /: There is a (potential) cause, but it is by itself 
insufficient, it can become fully effective as a cause only 
if a set of other conditions are given (“other conditions” 
in fig. 1). In the Pygmalion case, the teachers believe 
that some of their students have a particular aptitude 
which makes them “bloomers” (they will bloom up 
quickly), i.e., they will develop particularly fast. This apti- 
tude by itself is insufficient since without teaching it will 
not become effective. 

(b) N. This (potential) cause is a non-redundant part 
of the full “package” of influences; this means that the 
“package” will not become effective unless the (poten- 
tial) cause is an integral part of it. In “Pygmalion”, the 
“package” contains the aptitude mentioned above and 
appropriate teaching. The teachers assume that teaching 
“bloomers” will be more successful than teaching other 
students. 

(c) U. This “package” is unnecessary, there might be other 
“packages” which have a similar influence without the first 
“package” being present or with the first “package” or ele- 
ments thereof being present, but not the (potential) cause. In 
“Pygmalion”, one might argue with Rosenthal and Jacobson 
that it is not the aptitude of the students which triggers the 
acceleration in student development, but rather the expecta- 
tion of the teachers which makes them behave differently to- 
wards the potential “bloomers”. 

(d) S. This “package” is sufficient, if the full “package” is 
provided, the effect is likely to happen, whichever other con- 
ditions may occur. The teachers assume in the “Pygmalion” 
case that indeed it is the blooming potential - and not their 
expectation and the related behavior - that accounts for the 
acceleration. 

One can see in the Pygmalion example that using the INUS 
condition model permits to include biases in the explanation, 
e.g., in the Cause 2 (c component in fig. 1) and to provide an 
alternative explanation of the presumed effect. 

3 It could be formulated in a positive manner. It would read then: 
“A cause in an essential part of package of conditions which 
inevitably leeds to the effect, but this effect can also have 
other sauses than the one under investigation.” 
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Sis aiskinimo modelis daug sudetingesnis nei tra- 
dicinis aiskinimas, pvz., remiantis Hempel-Oppenheim 
schema. Bet galvojant apie socialinio elgesio ir veiks- 
mu sudetingumq, „INUS“ modelis atrodo tinkamesnis, 
leidziantis jtraukti daugiau butinq veiksniq, lemianciq 
rezultatq. Taciau tie jvairus veiksniai galetu buti susieti 
su jvairiomis teorijomis. Pvz., auksciau pateiktame pa- 
vyzdyje teorija j kuriq destytojai sutelkia demesj, yra 
gabumij teorija (1 priezastis, „N“ - elementas); taciau 
kitos teorijos taip pat svarbios, pavyzdziui, destymo 
teorija (kitos sqlygos, „l“ - elementas), eksperimen- 
tuotojo poveikio teorija (2 priezastis, „U“ - elementas) 
ir panasiai. Vadinasi, socialiniq tyrimq studijoje reike- 
tq kiek jmanoma issamiau paaiskinti tarn tikro „jeigu 
tai“ sqrysio „INUS“ modelio sqlygas, kur „jeigu“ kom- 
ponentas zyrmi (potencialiq) priezastj, o „tai“ kormpo- 
nentas - rezultatq. Is to isplaukia, kad teorineje pra- 
nesimo dalyje jvairios „INUS“ modelio sqlygq dalys 
turetu buti kiek jmanoma aiskiau isdestytos. 

Kaip pazymi Shadish, Cook, Campbell (2002, p. 5), 
„mes retai zinome visus veiksnius ir tai, kaip jie susijp 
vienas su kitu“. Ypac tai pasakytina apie praktinius tyri- 
mus (kurie yra tipiniq tyrimq ir kokybes uztikrinimo jver- 
tinimq pagrindas), kur nejmanoma paaiskinti visu po- 
tencialiq „INUS“ modeliu sqlygojamu veiksniq. Be to, 
lemiamas veiksnys tai, kad teorijos bet kuriuo atveju 
yra tikroves abstrakcijos, kur neatsizvelgiama j kai ku- 
riuos veiksnius, kurie galetu buti lemiami (taip pat zr. 
Herrmann, 1979, abstraktumo ir konkretumo dilemq, 
paminetqjvade). Nepaisantto, INUS modelis turetu buti 
naudingas. 

4 pasiulymas. Uzuot bandzius suprasti po vienakin- 
tamqji atskirai, reiketg isanaiizuoti sistemq remiantis 
INUS modeliu, t.y. meg inti nustatyti kiek jmanoma dau- 
giau potenciatiLj INUS veiksniu (ypac b dalyje: rinkinio 
elemental, kurie nera demesio centre, bei c daiyje: ki- 
tas potenciaius priezascig rinkinys)! 
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DAUGIALYPIAI METODAI 

Vienas ankstyvqju kritinio multiplizmo modeliq yra dau- 
giasavybe daugiarmetode matrica. 

3.1. CAMPBELL’O IR FISKE’O DAUGIASAVYBE 
DAUGIAMETODE MATRICA 

Donald T. Campbellas (1916-1996) gali buti laikomas 
kritinio multiplicizmo pradininku, nors ir nera pasiO- 
Igs sio termino, kuri reiketq sieti su vienu is Camp- 
bello studentq Thomu Cooku (pvz., 1985). Campbel- 
las rase apie trianguliacij^, o svarbiausiu cia laikoma 
Campbello ir Fiske’o (1959) darbas apie daugiasa- 
vybg daugiametodp matricq (angl. muititrait-muitimet- 
hod matrik). Remiantis siuo modeliu kiekvienas is ke- 
liq, tiketina, kad nepriklausomq, konstruktq (kuriuos 
autoriai vadina savybemis; C r C 2 , C 3 , ir t.t.), vertina- 
mi taikant jvairius metodus (M 1 , M 2 , M 3 ) ir vis jtrau- 
kiant tuos pacius subjektus. Tuomet rezultatus gali- 
ma sieti tarpusavyje ir isskirti jvairius pagrjstumo (tin- 
kamumo) tipus: 

• Konvergencinis pagrjstumas: koreliacija tarp to 
paties konstrukto (C : ) vertinimu taikant skirtingus me- 


This approach to explanation is much more complex 
than the traditional explanation schema, e.g., according to 
the Hempel-Oppenheim-schema. But looking at the com- 
plexity of social behavior and action, the INUS concept 
seems much more appropriate, giving the possibility to in- 
clude many factors required for an effect to occur. The dif- 
ferent factors, however, may be related to different theo- 
ries. E.g., in the example above, the theory the teachers 
are focusing on is a theory of aptitude (Cause 1, “N”-ele- 
ment); however, other theories are relevant as well, such 
as a theory of teaching (for the other conditions, T-ele- 
ment), the theory of experimenter effects (for the Cause 2, 
“U”-element), and the like. Within a social research study, 
hence, it would be appropriate to account as fully as pos- 
sible for the INUS conditions of a particular if-then-relation- 
ship, the if-component being the (potential) cause, the then- 
component being the effect. This means, in the theoretical 
part of the report, the different parts of the INUS conditions 
should be made explicit as much as possible. 

However, as Shadish, Cook and Campbell (2002, p. 
5) have pointed out, “we rarely know all of (the factors) 
and how they relate to each other”. Particularly in field 
settings - the typical settings in research and evaluation 
of quality assurance - it is impossible to account for all 
potential factors within a set of INUS conditions. This is 
due, among others, to the fact that theories are neces- 
sarily abstractions of reality, neglecting some factors 
which might be crucial (see also Herrmann’s, 1979, ge- 
neralizability-concreteness dilemma mentioned in the in- 
troduction). Nevertheless, it is appropriate to think in 
INUS terms. 

Suggestion 4: Instead of conceiving one independ- 
ent variable at a time, make an analysis of the system in 
INUS terms, i.e., try to identify as many potential INUS 
factors (particularly in the parts b: elements of the pack- 
age which are not focused, and c: other potential cause 
package)! 
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MULTIPLE METHODS 

One of the earliest approaches used in critical multiplism 
is the multitrait-multimethod matrix. 

3.1. THE MULTITRAIT-MULTIMETHOD MATRIX BY 
CAMPBELL AND FISKE 

Donald T. Campbell (1916-1996) can be seen as the founder 
of critical multiplism, although he did not coin the term him- 
self which goes back to one of his students, Thomas Cook 
(e.g., 1985). Instead, Campbell wrote about triangulation, 
the most important publication in this regard being Campbell 
and Fiske (1959) on the multitrait-multimethod matrix. Ac- 
cording to this approach, each of several presumably inde- 
pendent constructs (which the authors call “traits”; C r C 2 , 
C 3 , etc.) are assessed with different types of methods (M,, 
M 2 , M 3 ) with the same subjects. One can then correlate the 
results and distinguish different types of validity: 

• Convergent validity: Correlation between assessments 
of the same construct (Q) with different methods (M^ M 2 , 
etc.); for good assessment tools, this correlation should 
be as high as possible, the upper limit being in the range 
of the respective reliabilities. 
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todus (M,, M 2 ir t.t.); ieskant tinkamij vertinimo priemoniq 
si koreliacija turi buti kiek jmanoma aukstesne tai auks- 
tesnei reiksmei isliekant atitinkamo patikimumo (tikslumo) 
ribose. 

• diskriminantinis pagrjstumas I: koreliacijos tarp skirtin- 
gu konstruktLj (C,, C 2 , t.t.) vertinimq taikant tq patj metodq 
(M^; ieskant tinkamq vertinimo priemoniq si koreliacija turi 
buti zema-jei ji auksta, tai konstruktai nera nepriklausomi ir/ 
arba taikant metodq (MJ gaunami artefaktai. 

• diskriminantinis pagrjstumas II: koreliacijos tarp jvai- 
riq konstruktLj ( C v C 2 , t.t.) vertinimq taikant skirtingus me- 
todus (M r M 2 , etc.); ieskant tinkamu vertinimo priemoniq 
si koreliacija taip pat turi buti zema, jei ji auksta (tokia auksta 
kaip I diskriminantinio pagrjstumo atveju), tai rodo, kad kon- 
struktai nera nepriklausomi. 

Patry (1989) yra pasiulqs bendresnp koncepcijq, ap- 
imanciq ne tik vertinimo („priklausomas kintamasis"), bet 
ir modelj bei nepriklausomq kintamqjj. 

Koreliacijos gali buti jtrauktos j koreliacijos matricq, is 
cia atsiradps pavadinimas „daugiasavybe ir daugiameto- 
de matrica“ (sutrumpintai DSDM; angl. MTMM). Sis kon- 
vergencinio ir diskriminantinio pagrjstumo nustatymo me- 
todas tapo vienu is veiksmingiausiq vertinimo pagrjstumo 
mechanizmo priemoniq. Taciau atsizvelgiant j daugybp kri- 
tiniq pastabij, issakytq jvade, norstyrimai pagal DSDM daz- 
ni, tik keletu atvejq neabejotinai pavyko nustatyti ir konver- 
gentinj, ir diskriminantinj pagrjstumo (Fiske, 1982). Labai 
daznai I diskriminantinio pagrjstumo (tas pats metodas, bet 
skirtingi konstruktai) koreliacijos per aukstos, o tai rodo, 
kad butent metodai lemia aukstq kintamumo procentq ver- 
tinimuose; kitaip tariant, rezultatus sqlygoja taikomi meto- 
dai, o ne tiriamas reiskinys. Tad dar kartq jstikiname, kad 
kritinis multiplizmas neissprendzia problemu, bet greiciau 
atskleidzia (diagnozuoja, nustato) problemq. 

5 pasiOlymas. Taikykime DSDM! 

3.2. SALISKUMAS VERTINIMUOSE 
Taciau diagnozavimas yra butina sqlyga problemoms sprps- 
ti, jeigu toks sprendimas nera atsitiktinis arba „aklas“ (tai 
ganajdomi mintis, zr. Campbell, 1960, betvisiskai neeko- 
nomiska). Remiantis 1 pasiu/ymu, butq neteisinga neatsi- 
zvelgti j daznq vertinimo saliskumq, taciau reiketu paies- 
koti geresniq metodLj. 

Si diagnoze tik parodo, kad rnusq vertinimai labai sa- 
liski ir atskleidzia tokio saliskumo pobudj, bet nenurodo, 
kode ! jie tokie saliski. Todel mums reikalinga tinkama ver- 
tinimo teorija. 

Egzistuoja keletas teorijq, kurios galetu buti pasitelk- 
tos Siam tikslui. Christensen (1981) yra pasiulps „subjek- 
to motyvij paaiskinimo": vertinimo situacijose subjektai 
linkp maksimaliai padidinti kitiems zmonems daromp jspu- 
dj, t.y., jie siekia pozityvaus „savps pristatymo“ („saviro- 
dos“, angl. self-presentation). Patry (2004) sip spvoko is- 
plete pritaikps savo sukurtp situacijos specifiskumo teori- 
jp. Jis pasiule algoritmo, kuriuo gali buti pasiremta sie- 
kiant nustatyti, ar galima laukti „savps pristatymo" salis- 
kumo konkrecioje vertinimo situacijoje, ar ne. Sj algorit- 
mo matome 2 paveiksle. 

Svarbiausia cia tai, kad jei subjektas zinos, jog yra ste- 
bimas, sis zinojimas veiks jo elgesj (Patry, 1982). Keletas 
faktoriu gali padidinti (arba sumazinti) tikimybp, kad sub- 
jektas zino, jog jis yra stebimas. Visu pirma svarbios yra 
atviros (eksplicitines) instrukcijos. 


• Discriminant validity I: Correlations between as- 
sessments of different constructs (C,, C 2 , etc.) using 
the same method (M : ) ; for good assessment tools, this 
correlation should be low - if it is high, this indicates 
that the constructs are not independent and/or the used 
method (M.) produces artifacts. 

• Discriminant validity II: Correlation between as- 
sessments of different constructs (Cl, C2, etc.) using 
different methods (M r M 2 , etc.); for good assessment 
tools, this correlation should also be low - if it is high 
(as high as in discriminant validity I), there is indication 
that the constructs are not independent. 

Patry (1989) has proposed a more general ap- 
proach, including not only assessment (the “dependent 
variable”), but also the design and the independent vari- 
able. 

The correlations can be put in a correlation matrix, 
hence the name “mutitrait-multimethod matrix” (in 
short: MTMM). This method of testing both for conver- 
gent and discriminant validities has become one of 
the most powerful tools for validation of assessment 
instruments. In accordance with the many criticisms 
mentioned in the introduction above, however, despite 
frequent studies with MTMM, only few have been re- 
ally successful in showing both convergent and dis- 
criminant validities (Fiske, 1982). Very frequently, the 
correlations of discriminant validity I (same method, 
different constructs) are too high, which indicates that 
the methods account for a high percentage of vari- 
ability in the assessments; in other words, the results 
are due to the methods used, not to the phenomenon 
under investigation. Once again, it turns out that criti- 
cal multiplism is not the remedy, but rather the diag- 
nosis of the problem. 

Suggestion 5: Use MTMM! 

3.2. BIASES IN ASSESSMENT 
But the diagnosis is a necessary condition for the remedy 
if the latter is not to be random or “blind” (which might be 
an interesting idea, see Campbell, 1 960, yet not economic 
at all). According to suggestion 1, it would not be appro- 
priate to close the eyes with respect to the frequent as- 
sessment biases, but rather to seek for better methods. 

However, this diagnosis is only showing thatX here 
are strong biases in our assessments (and what kind 
of biases these might be), but gives no indication as to 
why there are biases. For this we would need an ap- 
propriate assessment theory. 

There are some theories which can be used for this 
function. Christensen (1981) has provided a “parsimo- 
nious explanation of subject motives”: In assessment 
situations, subjects tend to attempt to maximize the im- 
pression they make upon other people, i.e. , they aim at 
a positive self-presentation. Patry (2004) has extended 
this concept, using his theory of situation specificity, 
and has proposed an algorithm which can be used to 
detect whether one can expect a self-presentation bias 
in a given assessment situation or not. This algorithm 
is presented in figure 2. 

The most important issue here is that if the subject 
knows he or she is being observed this will have an im- 
pact on his or her behavior (Patry, 1 982). Several factors 
may increase (or decrease) the likelihood that a subject 
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2 pav. Pozymiu ir „savps pristatymo" (savirodos) tendenciju sarysis (Patry, 2004, p.46) 

Figure 2: Relationship between the attributes and the self-presentation tendency (from Patry, 2004, p. 46) 


Siose instrukcijose eksperimentuotojas nurodo sub- 
jektui, kqjisturi daryti (pvz., atsakyti j klausimyne esan- 
cius klausimus, valdyti tam tikrq mechanizing arba net- 
gi elgtis vienu ar kitu budu); sitaip eksperimentuotojas 
apribojasubjekto naturalu elgesj (Tunnell, 1977), osub- 
jektas tegali tik paklusti, jeigu jis tikrai zino apie tokj 
apribojimq, t.y. supranta, kad yra stebimas. 

Aplinka (patalpos ir kitos situacijos charakteristikos, 
kurios skiriasi nuo naturalios aplinkos; zr. Patry, 1982), 
kurioje vyksta stebejimas, taip pat turi jtakos. Jeigu sub- 
jektas prasomas eiti j tam tikras patalpas, kur jis bus 
stebimas, pvz., j eksperimentatoriaus laboratory^, sub- 
jektas negales nesuvokti, kad bus stebimas. Pakanka 
tik nezymiai pakeisti aplinkq ir subjektas suvoks, kad 
jis stebimas; pavyzdziui, tokj poveikj gali tureti patalpo- 
je esanti kamera (tai nustatyta remiantis stebetq asme- 
nu pastabomis, pvz., Ischi, 1978), net jeigu subjektas ir 
neturejo pereiti j kitq vietq del tokio tyrimo. 

Galima butij nurodyti ir daugiau veiksniq, taciau 
auksciau pamineti laikytini svarbiausiais. Vis delto yra 
budq siems veiksniams apriboti. Pirmiausiai instrukci- 
jas galima nukreipti; viena is galimybiu - pasinaudoti 
apgaule (priversti zmones patiketi, kad bus stebima 
kazkas kita, o ne tai, kas tikrai domina; zr. pvz., Miller, 
1972). Nors apie etines apgaules problemas kalbeta 
(pvz., Baumrind, 1985), tai nerataip problemiska kaip 
manyta (Christensen, 1988; P. Patry, 2001). Pagrindi- 
ne apgaules prielaida yra ta, kad zmonems suteikus 
klaidingq informacijq apie tyrimo tikslq, jie elgsis tarsi 
butLj negavp nurodymLj, susijusiij su svarbiais elgse- 
nos kintamaisiais. Taciau nors implicitiniq instrukciju 
kokybe ir kryptis butLj pakeista, tos implicitines instruk- 
cijos vis vien isliktu. Taigi galima teigti, kad Milgram 
(1963/1972) paklusnumo eksperimente apgaule pa- 
keite implicitines instrukcijas is „nebuk paklusnus“ (kas 
butp spontaniska savgs pristatymo strategija tiems, ku- 


knows that he or she is being observed. First of all, the ex- 
plicit instruction is relevant. This is the instruction an experi- 
menter gives to the subject about what to do (e.g., to re- 
spond to a questionnaire, to manipulate some machine or 
even to behave in a certain manner); by this, the experi- 
menter restricts the subject’s natural repertoire of behavior 
(Tunnell, 1 977), and the subject can only comply if he or she 
is indeed aware that such a restriction is taking place - which 
means he or she knows about being observed. 

A further influence is the setting (the room and other 
characteristics of the situation which differ from the natural 
situations; see Patry, 1982) of the observation. If the subject 
is asked to go to a certain place in order to be observed - 
e.g., to go to the experimenter’s laboratory -, the subject 
cannot avoid being aware of the observation. It is sufficient 
to make a slight change in the setting to remind the subject 
of the fact that he or she is being observed; for instance, the 
mere presence of a camera in the room may have this effect 
(as can be deduced from remarks of the observed persons, 
e.g., Ischi, 1978), even if the subject did not have to move to 
some other place for the purpose of the study. 

One could add more factors, but those seem to be the 
most important. However, there are some means to influ- 
ence them. First of all, one can re-direct the instruction; 
one possibility is deception (making people believe that 
something else is being observed than what is actually of 
interest; see, e.g., Miller, 1972). Although ethical problems 
of deception have been discussed (e.g., Baumrind, 1985), 
it seems less problematic than usually assumed (Christen- 
sen, 1988; R Patry, 2001). The basic assumption of decep- 
tion is that people who are given wrong information about 
the research topic will act as if uninstructed with respect to 
the relevant behavior variables. However, although the qual- 
ity and direction of the implicit instruction may have been 
altered, the implicit instruction itself is not removed. So one 
could argue that in Milgram’s (1963/1972) obedience ex- 
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rie zino apie Milgram tyrimo tikslq) j „pasirodyk esqs ge- 
ras subjektas" (Orne, 1962, prasme) arba „pasirodyk 
esqs patikimas subjektas" (Fillenbaum, 1966, prasme) 
ir panasiai, o tai sudaro dazniausiq mokymo(si) ekspe- 
rimento motyvq. Taciau subjektas vis vien gali jtarti eks- 
perimentatoriaus ketinimus (McGuire, 1969), tad visa tai 
implicitiniam instruktavimui suteikia kitokj atspalvj. 

Naudojantis klausimynais butinas tam tikras eksplici- 
tinis instruktavimas; vienas is budq „paslepti“ pradinj tyri- 
mo tiksl^- pasitelkti „subtilias“ priemones, t.y. tokias prie- 
mones, kurios akivaizdziai neparodytLj kokiam tyrimui jos 
skirtos (pvz., zr., Timbrook ir kt., 1993). Tai labiausiai sie- 
tina su tomis vertinimo priemonemis, kurias taiko Rors- 
chach - cia nurodymai sqmoningai neapibrezti (Wallace, 
1966), o tai gali sqlygoti zemq sio metodo pagrjstumq 
(Groth-Marnat, 1990, p. 280). 

Socialineje psichologijoje israsta daugybe budq, kaip 
pakeisti implicitinius nurodymus. Taikant kitus metodus 
meginta isvengti implicitinio instruktavimo. Keturi ypac 
daznai naudojami budai apima vadinamqjq naturalising 
aplinkq, pripratimq prie aplinkos, jtraukimq arba laisvq ne- 
varzomq elgesj (Csikszentmihalyi, 1975) ir elgesio tapi- 
mq automatisku arba rutinisku. 

Remiantis 2 pav. pavaizduotu modeliu pagrindinis 
vertinimo saliskumo saltinis siejamas su subjekto tiks- 
lais vertinimo situacijoje. Tai ypac svarbu kai vertinamas 
jvertinimas, nes socialiniai partneriai jvertindami visuo- 
met turi tam tikrq interesu, kadangi tokios studijos tiks- 
las yra jvertinti esamq padetj ir numatyti butinas inter- 
vencijas. Sios intervencijos is tiesq galetu stipriai paveikti 
subjektq, kurio elgesys vertinamas. Taigi skirtingai nei 
tyrimai, kur subjektai nesusiduria su svarbiomis savo el- 
gesio pasekmemis, jvertinimq metu subjektai labai ge- 
rai zino apie vertinimo poveikj, tad tiketina, jog turi siuo 
atzvilgiu savo tikslq. 

Sis modelis visiskai suderinamas su „INUS“ kon- 
cepcija ir daugialypemis teorijomis. Pirmiausiai tiriama 
priezastis yra tam tikra charakteristika, kuriq norima jver- 
tinti. Pavyzdziui, jei ketinama vertinti destymo kokybq 
ir siuo tikslu atliekama studentq apklausa (pvz., Patry 
ir Gastager, 2004 panaudotas pavyzdys), tai charakte- 
ristikos, kurios bus vertinamos, gali apimti psichologi- 
nj klimatq paskaitos metu, destytojo geranoriskumq, is- 
sukiu laipsnj destant, apmqstymu skaiciq auditorijoje, 
tai, ar tema suprantama, ar ne ir pan. Tai „ 1 priezastys" 
INUS modelyje. Teorijos nuostata yra tokia, kad stu- 
dentq atsakymai j siuos klausimus (3 pav.), suteiks zi- 
niq apie auditorijoje vyraujantj psichologinj klimatq, t.y. 
kad atsakymai bus pagrjsti. Bet ji („1 priezastis") bus 
veiksminga tik tuo atveju, jei kitos sqlygos patenkina- 
mos, pavyzdziui, ar studentai perskaito ir supranta klau- 
simq ir pan., o tai jau kitos teorijos klausimas. Be to, 


periment the deception turned the implicit instruction from 
“don’t be obedient” (which would be the spontaneous 
self-presentation strategy for someone who knows about 
Milgram’s research purpose) to “show that you are a 
good subject” (in the sense of Orne, 1 962) or “show that 
you are a faithful subject” (in the sense of Fillenbaum, 
1 966) or the like, which are the likely motives in a learn- 
ing experiment. Then again, the subject may be suspi- 
cious of the experimenter’s intent (McGuire, 1969), which 
adds another color to the implicit instruction. 

The use of questionnaires necessarily requires some 
explicit instruction; a way to “hide” the initial intention of 
the study is to use “subtle” items, i.e. items which do not 
show obviously what they are supposed to measure (see, 
for instance, Timbrook et al., 1 993). This is most extreme 
in assessment instruments like the Rorschach where the 
instruction is deliberately ambiguous (Wallace, 1966) - 
which may have contributed to the low validity (Groth- 
Marnat, 1990, p. 280f.) of this method. 

Social psychologists have been very creative in in- 
venting techniques to alter the subjects’ implicit instruc- 
tion. Other approaches have been used to avoid an im- 
plicit instruction altogether. Four such influence types 
have been used particularly often: the so-called natural- 
istic situation, habituation, involvement or flow (Csikszent- 
mihalyi, 1975), and rendering the behavior automatic or 
routine. 

According to the model of figure 2, the central source 
of assessment bias, hence, is linked with the goals the 
subject has in the assessment situation. This is particu- 
larly important in evaluation assessment because the 
stakeholders have always specific interests in evaluation 
situations since the aim of an evaluation study is to value 
the current state and to draw interventions from this - and 
these interventions might well have a strong impact on 
the subject whose behavior is being assessed. So unlike 
research where the subjects are not faced with important 
consequences for their behavior, in evaluation, the sub- 
jects are very much aware of the impact of the assess- 
ment and are likely to have goals in this respect. 

This approach is fully compatible with INUS and with 
the use of multiple theories. First, the cause under investi- 
gation is the characteristic one wants to assess. For in- 
stance, one wants to assess teaching quality, and for this 
one questions the students (e.g., as used by Patry & 
Gastager, 2004). Here the characteristics to be measured 
are, for instance, the climate in class, the kindness of the 
teacher, the degree of challenge in teaching, the amount 
of reflection in class, whether the topic can be understood 
or not, and the like. These are “Causes 1” in the INUS 
model. The theory, then, is that the students’ answers to 
items like the one presented in figure 3 will inform about 


PSICHOLOGINIS KLIMATAS AUDITORIJOJE 



(Studentg tarpusavio atm os 1 era, gera savijauta ir kt.) 

LABAI BLOGAS / 

LABAI GERAS / 

CLIMATE IN CLASS 

VERY BAD 

VERY GOOD 

(. Athmosphere among students, well-being, etc.) 




3 pav. Klausimyno elemento, skirto vertinti destymo kokybg, pavyzdys 

(isversta is Arbeitsgruppe "Evaluation von Lehrveranstaltungen an der Universitat Salzburg”, 2001) 

Figure 3: Example from an item to assess teaching quality 

(translated from Arbeitsgruppe "Evaluation von Lehrveranstaltungen an der Universitat Salzburg”, 2001 ) 
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studentams turetu buti suformuluoti tam tikri uzdavi- 
niai - pavyzdziui, kad jie privalo stengtis buti nuosir- 
dGs. Kadangi pateikiamas klausimynas, naudojamos 
ne tik eksplicitines, bet ir implicitines instrukcijos, tad 
akivaizdu, jog vertinama destymo kokybe - sito ne- 
jmanoma nuo studentq nuslepti. 

Taciau studentai gali tureti ir kitq tikslq. Pavyzdziui, 
jie gali nenoreti jzeisti destytojo pateikdami pernelyg 
blogus jvertinimus. 1 paveiksle sj tikslq vaizduotu 2 prie- 
zastis. Arba „apgaudinejamo destytojo" (zr. Patry ir Gas- 
tager, 2004) priezastis gali tureti jtakos ir studentai pa- 
sielgia taip, kaip liepe destytojas. Tuomet atsiranda dar 
vienas nurodymas (tiesioginis arba netiesioginis), ku- 
ris gali buti priesingas arba sutapti su pradiniu nurody- 
mu („buk nuosirdus"). 

DSDM tyrimai parade, kad metodas (pvz., stebeji- 
mas, o ne klausimynai) turi dideles jtakos vertinimui. I 
diskriminantinio pagrjstumo koeficientai (jvairios savy- 
bes, vienas metodas) turi tendencijq buti aukstesni uz 
tinkamus, o tai rodo, kad nors tiriamos charakteristikos 
(„savybes“) ir skirtingos, rezultatai liekatokie patys. Tuo 
tarpu konvergencinio pagrjstumo (viena savybe, skir- 
tingi metodai) koeficientai yra zemesni negu tikimasi: 
tq paciq charakteristikq („savybq“) vertinant skirtingais 
metodais, rezultatai labai skiriasi. Tai rodo, kad meto- 
das saliskas. 

Campbell (1969) poziuriu, 2 priezastis tureti) buti ti- 
ketina konkuruojanti hipoteze. Apibudinimas „tiketina“ 
teoriskai pagrjstas. Egzistuoja be galo daug jmanomij 
„2 priezasciq" ir visi) jq isaiskinti nejmanoma. Turetume 
atsizvelgti i tiketinas — ir tai butq teorija, kuri suteikiaziniq 
apie konkuruojanciu hipoteziq tiketinumq. Auksciau is- 
destyta teorija, pasitelkiant „savgs pristatymq" („saviro- 
dq“) ir tikslo strukturq, yra viena jmanomu teorijq, ji atro- 
do gana patikima, nes yra patikrinta empiriskai. Taciau 
pasitelkus kitas teorijas butu galima svarstyti kitokias ti- 
ketinas konkuruojancias teorijas (hipotezes). 

Saliskumo vertinimuose, pasiremdami tikimybe ir 
auksciau aptartu modeliu arba algoritmu, o ypac sio 
modelio apimamq tikslq svarbumu, galetume pateikti 
siqformuluotq: 

6 pasiOlymas. Vertindami kruopsciai isanaiizuokite 
visus jmanomus tiks/us, kuriuos gali tureti subjektai 
(pvz., „sav$s pristatymo", „savirodos“) irsig tiks/u po- 
veikj vertinamam e/gesiui bei pasitelkus daugiaiypius 
vertinimus nustatykite gaiimg saiiskumq! Kai kuriais at- 
vejais gaii netgi pavykti isvengti sio sa/iskumo, bet tuo- 
met butina atidziai isnagrineti, ar nekyia kitokio pobu- 
dzio saiiskumo pavoju arba kitu probiemg (pvz., etiniu 
probiemg). 

4 

APIBENDRINIMAS 

Siame straipsnyje galejau aptarti tik keletq kritinio mul- 
tiplizmo taikymo atvejq: daugialypes teorijas ir daugia- 
iypius veikimo principus. Svarbiausias kritinio multip- 
lizmo teorijos siulymas - taikyti apm^stytus daugiaiy- 
pius modelius ivairiuose lygmenyse. Taciau keletas kliu- 
ciu gali kelti pavojq kritinio multiplizmo taikymui. 

Teorijq pliuralizmas daugeliui mokslininkq lieka svar- 
biausia problema. Jei Kuhn (1962) darbq interpretuo- 


classroom climate, i.e., that the answers are valid. But this 
works only if other conditions are satisfied, like the students 
can read and understand the question and the like - this is 
matter of another theory. Also, a certain goal structure must 
be given in the students: For instance, they must try to be 
honest. Since it is a questionnaire, there is an explicit as well 
as an implicit instruction, and it is obvious that the assess- 
ment deals with teaching quality - there is no way to avoid 
communicating this to the students. 

But the students might have other goals as well. For 
instance, they might not want to harm the teacher by 
giving too bad ratings. This goal then would be a Cause 
2 in figure 1 . Or a “cheating teacher” (see Patry & Gas- 
tager, 2004) may try to influence the students, and they 
want to behave like the teacher asks them to. Here, there 
is then another instruction (direct or implicit) which may 
or may not be in opposition to the initial instruction (“be 
honest”). 

In the MTMM research, it has become obvious that the 
method (e.g., observation as opposed to questionnaires) 
has a strong impact on the result. Discriminant validity coef- 
ficients I (hetero-trait mono-method) tend to be higher than 
appropriate, which means that even if the characteristics to 
be assessed (“traits”) are different, the results remain the 
same, whereas convergent validity (mono-trait hetero- 
method) are lower than expected: The same characteristic 
(“trait”), assessed with different methods, yields substantial 
differences. This would indicate a method bias. 

In terms of Campbell (1969), Cause 2 would be a plau- 
sible rival hypothesis. The term “plausible” refers, once 
again, to theory. Indeed, there are infinitely many possible 
“Causes 2”. It is not possible to account for all of them. 
One must consider those which are plausible - and it is 
theory which provides information on the plausibility of ri- 
val hypotheses. The theory presented above, using self 
presentation and the goal structure, is one of the possible 
theories which seems fairly plausible since it has been 
empirically tested. However, based on different theories, 
one might conceive other plausible rival hypotheses. 

From the likelihood of biases in assessment, the model 
or algorithm presented above and, in particular, the impor- 
tance of the goals within it, we can formulate 

suggestion 6: For any assessment, make a careful 
analysis of possible goals the subjects might have (e.g., of 
seif presentation), of the impact this might have on the as- 
sessed behavior, and of possibilities to detect biases 
through multiple assessments/ in some cases it might e ven 
be possible to avoid these biases, but then you must care- 
fully analyze if other biases or other problems arouse from 
this (e.g., ethical problems). 


4 

DISCUSSION 

In the present paper, I could deal only with a few uses of 
critical multiplism: multiple theories and multiple opera- 
tionalizations. The suggestion of critical multiplism in ge- 
neral is to use planned multiple approaches on different 
levels. There are, however, several obstacles which might 
jeopardize the use of critical multiplism. 

Pluralism of theories is a major problem for many sci- 
entists. If Kuhn’s (1962) work is interpreted as piece of 
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tume kaip aprasomosios sociologijos mokslo pavyzdj, tai 
jo aprasoma tai, kq mokslininkai suvokia kaip „gerq tyri- 
mq“, butent teorijq konkurencija. Siuo atveju teorijq pliura- 
lizmas laikomas nepriimtinu. Zinoma, patirtis, jgyta konfe- 
rencijq metu, skelbiant straipsnius (pvz., recenzuojamuo- 
se zurnaluose) ir projektq pasiulymq apzvalgq metu, rodo, 
kad socialiniuose moksluose, kaip ir daugelyje kitu moks- 
liniq discipline, teorijq gynimas arba jq nugincijimas yra 
svarbiausias mokslines kulturos elementas. 

Kaip jau mineta (pvz., Gastager ir Patry, 2000), prakti- 
nj darbq dirbantys specialistai, priesingai mokslininkams, 
sugeba patvirtinti priestaringas teorijas (pvz., kaip tradici- 
nio ir konstruktyvistinio mokymo teorijas) vienu metu. Ta- 
ciau sios teorijos nera tarpusavy susijusios. Prireikus, mi- 
neti specialistai gali pagrjsti savo praktinius jgudzius rem- 
damiesi bet kuria viena teorija (Patry, Schwetz ir Gasta- 
ger, 2000) ir atmesti kitas. Visa tai kertasi su mokslininkq 
darbo budais. 

Taciau ne vienas is minetu dviejq budq - nei abipuse 
atskirtis, kuria remiasi mokslininkai, nei vienas kuris nors 
konkretus panaudojimas atsizvelgiant j tarn tikrus porei- 
kius, kaip elgiasi praktikai - nera tinkamas. Kritinio multip- 
lizmo siekiamas teorijq pliuralizmas nereiskia nei teorijq 
konkurencijos, nei jq koegzistencijos (atskirose „ekologi- 
nese nisose“), bet papildomumq (galimybg jas papildyti). 
Bent jau turetu buti pasitelkta „INUS“ koncepcijos mode- 
lis, kuriuo remiantis pritaikomi daugialypiai galimi panasaus 
reiskinio paaiskinimai. 

DSDM yra vienas is budu visa tai gyvendinti: pasitelkus 
jvairius vertinimo metodus arba, remiantis Patry (1989), net- 
gi pritaikius skirtingus metodus tiek nepriklausomam, tiek 
priklausomam kintamajam nustatyti. Taciau tokie tyrimai 
brangiai kainuoja, o tai, matyt, yra viena svarbiausiq kliu- 
ciq taikyti kritinj multiplizmq. Turimi resursai (finansiniai is- 
tekliai, personalas ir kt.) paprastai nera pakankami naudo- 
ti kritinj multiplizmq. Paskutinis, 

7 pasiuiymas. Social in ig tyrimu finansavimas turetu buti 
gerokai didesnis, kad kritini multi pi izma butg gaiima taiky- 
ti. Tai svarbu ir studiju kokybes jvertinimo atveju. 

Vadinasi, atsiduriame tarsi uzburtame rate: socialiniq 
tyrimq kokybe abejotina (zr. iskeltas problemas jvade), to- 
del finansavimas menkas. O menkas finansavimas trukdo 
tyrimq tobulinim^, nors ir zinome, kaip juos tobulinti - pa- 
sitelkus kritinj multiplizm^. 


(descriptive) sociology of science, it describes what 
researchers perceive as “good research”, namely 
competition of theories. In this case, pluralism of theo- 
ries is seen as inappropriate. Indeed, experience in 
conferences as well as with publication practices (e.g., 
with reviewed journals) and project proposal reviews 
indicates that in social sciences - as in most other 
scientific disciplines - the defense or challenge of theo- 
ries is a major element of scientific culture. 

We have shown elsewhere (e.g., Gastager & Patry, 
2000) that practitioners, in contrast, have no problem 
in sustaining contradictory theories (like theories of tra- 
ditional and constructivist teaching) simultaneously; 
however, these theories are not linked with each other. 
Upon request, the practitioners can provide practice 
according to either theory (Patry, Schwetz & Gastager, 
2000) and ignore the opposite one. This is in flagrant 
contrast to the researchers’ practice. 

However, neither of the two practices - mutual ex- 
clusion, as done by the researchers, or use of either 
one according to the specific needs, as done by the 
practitioners - is appropriate. Pluralism of theories as 
aimed at in critical multiplism means neither competi- 
tion nor coexistence (in distinct “ecological niches”) 
of theories, but it means complementarity. At the very 
least, the “INUS”-structure should be applied, which 
means that multiple potential explanations of a similar 
phenomenon should be applied. 

The MTMM is one possibility to do so: Using differ- 
ent methods for assessment or, according to Patry (1 989) , 
even using different methods both in the independent 
and in the dependent variable. However, this is very 
costly. And this is probably one of the most important 
obstacles to applying critical multiplism: The resources 
(financial means, personnel, etc.) are usually not such 
that critical multiplism can be used. Hence the last 

suggestion 7: Funding in social research should be 
much higher so that critical multiplism can be achieved. 
This holds for evaluation studies as well. 

Currently there is a vicious circle in effect: Social Re- 
search is of doubtful quality (see the problems in the first 
section), so there is little funding. And the little funding does 
not permit the research to improve, although we would know 
howto improve: through critical multiplism. 
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